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ABSTRACT 

This document reports an attempt to examine the 
content and structure of the domain of research and development (R&D) 
training resources. The project began by locating instructional 
materials within a larger matrix that classified R&D resources by 
structure (primary/sec ondary, oral/written, doc umentary/non- document) 
and level of formalization (formal, semif ormal, informal) , and 
content (instructional/non^instructional material) . Examination of 
ERIC, "Books in Print," and three nonbook instructional materials 
collections indicated that there is roughly an "order of magnitude" 
ratio for the estimated quantities in these three bases. 
Extrapolation suggests that 90% of the educational R&D instructional 
resources are largely informal and fugitive. Perhaps 10% reach a 
semiformal level where they may be in published form and may be 
referenced. Possibly only one or two percent are published 
commercially in hard cover, and much less than one percent are 
subjected to the test of the RSD development cycle. Probably over 95% 
of all instructional materials are in printed rather than oral form. 
The number of fully developed and tested materials is probably well 
under 100 items; perhaps another 1000 iteE^s have undergone minimal 
development and informal testing or evaluation. The number of books 
and textbooks with reasonably direct bearing on R&D content is 
estimated at 10,000. Appended to this document are scattergrams of 
counts; a listing of RIE descriptors; and estimations of numbers of 
documents in RIE, CUE, and "Books in Print." (Author/DDO) 
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EXECUTIVE SUMMARY 



In July 1972, the Far West Laboratory for Educational Research and 
Development initiated a project designed to develop and test procedures 
to evaluate and disseminate information on training of educational research 
and development (R&D) personnel. 

One aspect of the project called for the creation of matrices that would 
map the domain of R&D personnel and training information. As the project 
progressed, the sponsor*s interest focused particularly on training materials. 
This document reports the author's effort to establish data-based estimates 
for the subject content and to design a format of educational R&D training 
materials. 

After locating the area of specific investigation within the larger 
area of R&D information, estimates based on ERIC, Books in Print , and a 
combination of three instructional materials collections were examined in 
detail and compared to each other in 14 major subject-content categories. 
Examination of these data bases indicate there is roughly an "order of 
magnitude" ratio for the three bases. Extrapolation suggests that ninety 
percent of the educational R&D instructional resources are largely informal 
and fugitive. Perhaps ten percent reach a semi formal level where they may 
be in published form and may be referenced. Possibly only one or two per- 
cent achieve "hard cover" commercial publication and much less than one 
percent are subjected to the test of the R^jD development cycle. Probably 
over 95 percent of all instructional materials is in printed form. The 
number of fully developed and tested materials is probably much less thar. 
100 items; perhaps 1,000 have undergone minimal development and may be 
generally accessible although not necessarily easily so. And there .ire 
possibly 10,000 books and textbooks which bear on R&D training content as 
defined by the project. ERIC documentation suggests there are more than 
40,000 relatively recfent (1966-1972) journal articles and reports which 
provide the knowledge base for R&D training content. The total relevant 
published literature may be closer to 100,000 items. 

There are significant but far from perfect content correlations among 
the three data bases. The ERIC descriptor counts prove to be better 
predictors of counts of nonbook instructional materials subject categories 
than do count of Books in Print . Almost all the instructional materials 
are located in classical "educational research categories," with only three 
of eleven major content categories containing over half of the books 
and textbooks and over forty percent of the nonbook instructional materials. 
By contrast only four percent of all instructional materials were classified 
in the tv/o general content categories of (a) Design and Development and 
(b) Evaluation. However, examination of trends in ERIC as well as other 
evidence^ suggest that the raite of growth may be greater in these less 
well-established but high interest areas. 
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THE DOMAIN OF R&D TRAINING RESOURCES 
I. INTRODUCTION 

Training resources may be classified as actual or potential . Actual 
training resources are materials that have been specifically designed to 
serve instructional purposes such as textbooks, programmed workbooks, in- 
structional films, etc. Potential resources are materials which have not 
been designed primarily to serve instructional purposes, but which can be 
adapted to meet instructional needs, or which may be used as a knowledge base 
to create instructional materials. Examples are journal articles, case 
studies, technical reports, etc. There is no clear-cut boundary, except 
possibly the intent of the author. Certainly, a well-written journal article 
may have more instructive potential than a poorly written chapter in a text- 
book. 

Not only is there a problem in deciding what is an instructional material, 
there is also a problem in deciding what is an educational R&D instructional 
material. This raises the question of content. As we have noted elsewhere 
(Hood, July, 1973, p. 10), educational R&D has a "field-oriented" knowledge 
base, and derives its content from many disciplines. Consequently, the 
theory, methodology, art, and practice, which may constitute suitable content 
for instruction, extends into many areas apart from education. The determi- 
nation of the boundaries for educational R&D content must be somewhat arbitrary. 

II. THE DOMAIN OF R&D INFORMATION 
A' Structure and Levols of Formalization 

Because of the blurred distinctions regarding training resources and R&D 
content, it is useful to regard R&D training materials within the larger domain 
of R&D information. To do so, we shall review an earlier analysis (Hood, 
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McCutchan, Aldrich, and Wolf, 1972, pp. 70-77) in which structure , levels of 
formalization , and content were viewed as three major classification constructs. 

Structure . Information may have many forms and formats. Primary informa- 
tion may include oral comnuni cation, primary documents (e.g., reports, journal 
articles, or conference papers), and primary nondocument items (e.g., instruc- 
tional films or testing materials). Secondary information includes these 
sources: abstracts, annotations, bibliographic citations, indexed information, 
bibliographies, literature reviews, magnetic tape data, as well as directories, 
registers, and guidebooks (e.g., personnel directories, consultant registers, 
organizational directories, subject literature guides). 

A secondary aspect of structure is the format or medium in which the in- 
formation appears. Is it oral, documentary or nondocument in form? And, for 
each of these forms, what specific format does it have, e.g., if oral, is it a 
meeting, symposium, conference, seminar, telephone call, casual meeting, etc.? 

Level of Formalization . This dimension may be roughly divided into three 
levels: formal, semiformal and informal. These levels can be associated with 
timel iness , accessibility , permanence , quality , and flexibility . Of course each 
of these characteristics could be treated as a separate dimension, but in 
general 3 they are all related to level of formalization. Each level has its 
advantages and disadvantages which have been outlined in some detail (loc. cit.). 
B. Instructional Materials in the R&D Information Domain 

Figure 1 maps instructional materials within the domain of R&D informa- 
tion, categorizing them by structure and level of formalization. The heavy 
border^ including the entries for noninstructional forms (documents and non- 
documents), instructional forms, and references at the formal and the semi- 
formal levels, intlicates our view of the feasible boundaries of an RID informa- 
tion collection. The very heavy border around the formal/instructional form 
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cell indicates the probable boundary for an instructional materials collection. 
If an unusual effort were made, it might be possible to locate and acquire 
semiformal instructional forms. The dotted line within the semiformal/ 
instructional fom cell identifies the possible extension. 

Content as a Third Classification . Figure 2 introduces content as the 
third classification dimension in a schematic way. Note that we have simpli- 
fied the scheme suggested by Figure 1 by omitting all oral forms and reference 
forms, and by deleting the entire informal level of fcmialization. This 
matrix, relating content, form, and level of formalization, was used in 
considering the problem of quantitatively estimating both absolute and 
relative numbers of primary and secondary instructional materials. If an 
adequate sampling frame existed it would be a relatively straightforward task 
of first classifying each item (thereby assigning it to one of the cells in 
Figure 2), then enumerating items within cells and by rows, columns, and 
blocks. However, nothing approaching an adequate sampling frame now exists. 
Although we have now perused descriptions of over 1500 instructional materials, 
and literally tens of thousands of titles of books, serials, etc., we have 
not found, nor have we been able to create, a collection whose composition 
is not susceptible to criticism. Despite these problems, we persisted in 
estimating the content of the R&D domain. The following section describes 
the approach and the results. 
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FIGURE 2 

^ MATRIX: CONTENT BY STRUCTURE BY LEVEL 
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III. ESTIMATION OF AMOUNT OF TRAINING 
RESOURCE BY CONTENT CATEGORIES 

A. Three Data Bases 

In attempting to compare content of various information bases, ERIC, Books 

in Print, and three different instructional materials collections were examined. 

This led to the conception of a two-dimensional matrix (Figure 3) In which 

levels of formalization have been collapsed. The content of instructional 

materials was estimated by using two information bases: Subject Guide to 

Books in Print , 1972, and three instructional materials files (Guba and Gephart, 

1970; Houston, 1973; and the Far West Laboratory's collection). Content for 

★ 

noninstructional materials was estimated by the ERIC collection. 

There are marked differences in the sizes of these bases. The ERIC 
collection, in December, 1972, included over 128,000 items, nearly evenly 
divided between journal article references in the Current Index to Journals 
in Education (CUE) and references to reports and other documents in Research 
in Education (RIE). The Subject Guide to Books in Print , 1972, lists over 
265,000 titles. But we have only slightly more than 1500 references to non- 
book instructional materials. A somewhat different picture emerges when we 
estimate the number of items "relevant" to educational R&D personnel and 
training content in each base. There is roughly an order of magnitude differ- 
ence in the estimated size of the "R&D collections"; namely over 45,000 items 
in ERIC, over 6,000 books, and over 700 nonbook instructional materials. The 
bases for these estimates will be discussed later. 

Two other data bases were examined and rejected. Paperbound B c ^^ks in Print 
does list some titles not found in the Subject Guide to Books in Print , however, 
the majority of the titles are duplicate?; A variety of indexes and catalogues 
(e.g^, the NICEM Indexes and the Westinghouse Learning Directory ) were also 
examined. The proportion of possibly relevant items was found to be so tiny, 
and our decisions based on listed information were judged to be so unreliable 
that searches using these resources were abandoned in favor of usinc; the 
Houston, Guba and Gephart, and Far West collections. 

Q 
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FIGURE 3 

MATRIX: CONTENT BY INFORMATION BASE 





NON INSTRUCTIONAL 
MATERIALS 


INSTKUCTIONAL ^TERIALS 


CONTENT 


ERIC 


BOOKS 


i NONBOOK 


Policy... 


# 


# 




Research 




# 




Development 


# 


# 




Etc. 






1 


Estimated R&D Collection 


> 45, 000 


> 6,000 


700 


Total Size of Base 


>1 28,000 


> 265, 000 


i >1500 

i 
1 



Note: # indicates number to be estimated. 
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us approximately 35 percent of ERIC and 48 percent of the "other" in- 
structional materials collections were judged to be relevant for a very broadly- 
defined conception of educational R&D personnel and training needs. Given the 
nature of these two bases, these relatively high percentages are not surprising* 
Neither is It surprising that only slightly over two percent of the titles In 
Books in Print should be considered relevant. 

To estimate the number and content of actual instructional materials, only 
the Books in Print and the nonbook instructional materials collections are 
needed. We believe that comparison to the ERIC collection, while not providing 
direct information on instructional materials ger se is useful for two reasons. 
First, ERIC lists recent documentary activity— the eventual "raw material" for 
textbooks and instructional materials. Second, our information regarding the 
content of the ERIC base is decidedly more comprehensive and reliable than 
for the other two bases. The ERIC data thus provides a useful comparison. 
Our general objective was to first place data in the cells of a matrix such as 
Figure 3, and then to compare the three bases. 

Although the task of obtaining the data seemed feasible, it turned out 
to be a major effort requiring a number of arbitrary decisions and several 
''shortcuts'' to obtain usable data at a reasonable cost. Before examining 
actual figures, it may be wise to recount the major difficulties. (Details 
are presented in Appendices B, C, and D.) 
D. Estimate for ERI C 

Lockheed Information Sciences of Palo Alto, California, provided us with 
computer-generated frequency counts of ERIC descriptor usage based on RIE 
documents EDOOlOOl to ED066200 and EJOOOOOl to EJ062751. 
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This Lockheed Information Science's list contained over 5,000 descriptors. 
We selected 379 of these which are conceivably relevant to educational R&D 
personnel and training needs. The descriptors were organized into 14 major 
categories and were grouped in three clusters. (See Table 1.) The descriptors 
and the ERIC usage counts are provided in Appendix B. 



TABLE 1 

FREQUENCY AND RELATIVE USE OF CLASSES OF DESCRIPTOR 
TERMS RELATING TO EDUCATIONAL RDD&E PERSONNEL AND 
TRAINING TOPICS IN RIE/CIJE (uECEMBER 1972) 









Frequency 


- . 




No. of 


of Use 


Per- 


Topics 


Terms 


(Thousands) 


centage 


Policy, Planning, Change, Innovation 


(16) 


15.1 


8.2 




(34) 


18.2 


9.8 




(17) 


12.9 


7.0 


Dissemination and Communication 


(10) 


4.3 


2.3 


Evaluation 


(21) 


15.0 


8.1 










[35.4] 


Science and Technology .... 






2.8 


1.5 


Data Analysis, Statistics, Other Analyses . . . 






11.3 


6.1 


Objectives and Criteria 


( 


:ii) 


3.7 


2.0 


Tests, f^asurement. Data Collection Instruments. 




(66) 


26.7 


14.4 


Methodology, Techniquey Theory 


1 


[47) 


30.3 


16.4 


'k^k'k'k'k'k 






[40.41 




( .9) 


2.4 


1.3 


Skills* Training, Professional Education .... 


(28) 


11.5 


6.2 


Personnel 


(44) 


10.3 


5.6 




(36) 


20.4 


11.0 










[24.1] 


Total 


379 


184.9 


99.9 
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The entries in Table 1 can be read this way: Sixteen descriptors, 
classed under the heading, "Policy, Planning, Change, and Innovation" were 
selected. These 16 descriptors were used 15.1 thousand times--8.2 percent 
of the total use of the 379 descriptors listed in Appendix B. We have, in 
effect, "adopted" the ERIC Thesaurus of Descriptors as our basis for content 
classification and have selected the 379 descriptors listed in Appendix B 
as our "operational definition" of educational R&D training content, as 
defined by the ERIC descriptors. This "definition" is totally arbitrary and 
based solely on the author's judgment. Appendix B must be examined to under- 
stand what was considered. 

Given these descriptor counts, we needed to estimate the number of 
documents represented. Since the ERIC descriptors index "information" 
within documents, and since several kinds of information, each calling for 
a different descriptor, is contained in the same document, the number of 
documents represented must be substantially smaller than the descriptor 
usage counts. Our next problem was to find a way to convert usage counts to 
estimates of document counts. Appendix C describes our estimation method 
which resulted in an estimate that the ERIC collection (as of December, 1972) 
contained at least 45,000 items to which the descriptors listed in Appendix B 
have been applied. 

The first cluster "Policy..., Research, Design and Development and 
Evaluation," which may be considered the R&D content "core," is estimated 
to include 16,000 to 26,000 ERIC documents with slightly more than half in 
RIE and slightly less than half in CUE. The RIE collection dates back to 
1966 and CUE, to 1969. Since CUE started later and currently indexes 
nearly one and a half times as many items as RIE, the ratio between RIE 
documents and CUE documents may change substantially in the next few years. 
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If a reasonably comprehensive, retrospective search (1964-1973) were made of 
published materials (including all ERIC documents) pertinent to only this 
"Policy. . .RDD&E" set, the number of related documents (e.g., those on statistics, 
measurement, methodology, personnel, content and media, etc.) would be pro- 
portionally larger. 

This very large number of items which could be included in an educational 
RDD&E personnel and training document collection reminds us of Paisley's 
dictum (Hood, et al. , September, 1972, p. 86) that the information center 
should exist to disseminate information, not archive it, and that its 
collection should reflect users' needs, not the size of the knov/ledge base. 

To summarize, our estimation of journals and other documents, which may 
be considered as potential instructional materials or as a knowledge base for 
instructional materials, has been based entirely on ERIC, primarily because 
the base is large, its content descriptor system has been carefully developed, 
and statistics regarding its content are available. The possible flaws in 
the ERIC estimates are that the collection is recent and ERIC does not 
provide much information on commerical publications, like books. 

Most of the materials in ERIC can be located in Figure 1 in the cells 
for formal of semiformal noninstructional forms. The ERIC collection 
analysis, because it is based on over 128,000 documents, with at least 
45,000 of these relevant to RDD&E personnel and training topics, affords a 
useful and moderately reliable "yardstick" against which we may judge the 
results of our subsequent examination of instructional materials based on^ 
Books in Print and on the Guba and Gephart, the Houston, and the Far West 
Laboratory files. 
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Unfortunately, there is no comparable, direct basis for estimating 
informal or unreferenced semi formal materials comparable to the ERIC 
content. We assume that these less formal items have roughly the same 
relative frequencies for content. Obviously, the absolute number cf informal 
and semi formal documents will be many times the number of formal documents, 
C. Estimates for Books in Print 

Naturally, published textbooks and other professional books may serve 
directly as instructional resources. We used Bowker*s Subject Guide to Books 
in Print , 1972, which includes approximately 265,000 titles, as our base 
for estimating the number of pertinent books. In seeking relevant subject 
headings, 1826 pages had to be manually scanned and each title within a 
selected subject heading had to be inspected. This is a considerably more 
fallible analysis than was performed with RIE and CUE, which are relatively 
thoroughly indexed. Approximately six descriptors are applied to the average 
"RDD&E" CUE document and nine and one-half descriptors to the RIE document. 
By contrast, 265,000 books appear only 392,000 times (slightly less than one and 
a half times per book) in the Subject Guide to Books in Print . Moreover, the 
initial Bowker c1asi;ifi cation, and certainly our own classification, has been 
based primarily on the book title. Since not every entry on every page was 
read, undoubtedly many items which appeared under unusual subject headings 
have been missed. On the other hand, the counts have been "generous" but 
selective; nearly every book which seemed to bear on RDD&E or personnel and 
training content ms counted, although not all books in a category were 
counted. For instance, under the "Disciplines and Specialties" category, 
18 out of over 800 titles in ''Economics" and 10 out of 42 titles in 
"Psychology^-Experimental" were counted. Unlike the RIE/CIJE data, these 
are not computer-derived frequencies or descriptors, but only rough, highly 
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subjective judgments of whether bo.oks in selected subject listings should be 
considered as relevant or valuable. 

Although the ERIC descriptors and the Books in Print subject headings 
are often quite different, an attempt has been made to summarize the results 
under the same major subject headings used in Table 1. The data for Books 
in Print are displayed in Table 2. 

TABLE 2 



NUMBER OF TITLES IN BOOKS IN PRINT , 1972, 
RELATING TO EDUCATIONAL RDD&E PERSONNEL AND TRAINING TOPICS 





No. of 


Per- 


Topics 


Titles 


cent 




738 


10.7 




791 


11.5 




82 


1.2 




316 


4.6 




109 


1.6 










49 


0.7 


Data Analysis, Statistics, Other Analyses . . 


1127 


16.4 




147 


2.1 


Tests, Measurement, Data Collection Instruments 


617 


9.0 




1406 


20.4 










614 


8.9 


Skills, Training, Professional Education . • • 


169 


2.5 




280 


4.1 




435 


6.3 


Total 


6880 


100.0 



Nearly seven thousand titles were found, with over 2,000 titles in 
the fn-st "Policy. . .RDD&E" cluster. The great majority of these titles deal 
with "Policy, Planning, Change, Innovation," "Research," "Dissemination and Com- 
munication." Markedly underrepresente^i .iro books in "Evaluation" and in "Design 
and Development" (which excludes instructional technology, programmed instruction 
titles that appear under "Content and Media"). Looking for Design and Development" 
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or "Evaluation*' books in Books in Print proved to be like looking for the pro- 
verbial needle in a haystack- The Books in Print subject headings are not at all 
helpful. For instance, the Stuff lebeam, et al- PDK book on Educational 
Evaluation and Decision-Making was finally found under "School Management 
and Organization." The counts on "Evaluation" and "Design and Development" 
undoubtedly are underestimates, but are probably close. 

While we found less than 200 books in the "Design and Development" and 
in the "Evaluation" categories, we found nearly eight hundred books in the 
"Research" category, over eleven hundred books in the "Data Analysis, Statistics, 
and other Analyses" category and over fourteen hundred books in the "Methodology, 
Technique, Theory" category. 
D. Estimates for Nonbook Materials 

We combined three sources to estimate nonbook instructional resources. 
Resources for Performance-Based Education (Houston, et al., 1973) lists and 
annotates over 1,100 instructional materials including films, slides, tapes, 
modules, programmed texts, and multimedia kits. The materials pertain 
primarily to training prospective and inservice education personnel. 
Specifically excluded are textbooks, materials produced prior to 1967, 
resources for children, materials (unless they are in modular format) whose 
purpose is to teach elementary or secondary subjects, descriptions of 
programs, and lists of objectives or competences. 

Most of the materials are still being developed and many are undergoing 
revision. To compile their list, the authors had contactad performance-based 
education developers throughout the country, commercial publishers, R&D 
centers, educational laboratories, professional organizations, Teacher Corps 
projects, special education instructional materials centers, protocol training 
projects, university audiovisual centers and others concerned with development 
O d resources. 

ERIC 
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Guba and Gephart (1970) had made an earlier search which focused more 
directly on collecting instructional materials used in teaching about 
research processes • After a notably unproductive literature search and 
an unrewarding effort to solicit materials through announcements in pro- 
fessional journals and newsletters, they turned to direct contacts through 
visits, meetings, and correspondence. These contacts netted 327 different 
items including textual materials, learning exercises, bibliogra.ohies, 
examinations, course descriptions, etc. Virtually all of these materials 
could be classified as semiformal, although a few have since emerged in 
formal publications. 

The Far West Laboratory collection of instructional materials focuses 
primarily on educational development, diffusion, and evaluation, applied 
research, and educational policy, planning and management. Excluding pub- 
listed textbooks, it contains approximately one hundred additional items. 

The Far West collection was assembled in a number of ways including 
contacting R&D training materials developers, monitoring the CEDaR catalogues 
and other instructional materials catalogues. 

Taken together, these three sources list approximately 1500 different 
items. As in the case of our search of Books in Print , our counts for 
these other instructional materials were made manually. However, in this 
instance we worked from annotations rather than from titles only. Our counts 
are displayed in Table 3. 
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TABLE 3 

NUMBER OF NONBOOK INSTRUCTIONAL MATERIALS 
RELATING TO EDUCATIONAL RDD&E PERSONNEL AND TRAINING TOPICS 













• C 1 


Topics 


Count 


cent 


Policy, Planning, Change, Innovation 


79 


10.7 


Research .... 


71 


9.6 




44 


6.0 


Dissemination and Communication 


29 


3.9 


Evaluation 


40 


5.4 


iricicicicic 






Science and Technology 


5 


0.7 


Data Analysis, Statistics, Other Analyses . . . 


139 


18.9 


Objectives and Criteria 


62 


8.4 


Tests, Measurement, Data Collection Instruments 


65 


8.8 


Methodology, Technique, Theory 


79 


10.7 


****** 






Disciplines and Specialities 


11 


1.5 


Skills, Training, Professional Education .... 


28 


3.8 


Personnel 


5 


0.7 


Content and Media 


80 


10.9 


Total 


737 


100.0 



E . Comparison of the Thre e Bases 

We have noted in sections B, C, and D how we arrived at the counts 
listed in Tables 1, 2, and 3. These data are consolidated and compared in 
Table 4 on the following page; both the count and the percentage ratio of 
the count to its base is presented. The counts for books and for "other" 
instructional materials are counts of titles. The count for journals and 
other documents is an estimate derived by multiplying the ERIC descriptor 
percentages of Table 1 against an assumed base of 40,000 documents, each 
document to be assigned to only one of the 14 categories. 
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TABLE 4 

COMPARISON OF CONTENT COUNTS IN 
THREE INFORMATION BASES 







Instructional Materials 


ERIC Journals & 




Subject Content Category 


Nonbooks 


Books 


Other 


documents 








Per- 




Per- 




Per- 






No. 


cent 


No. 


cent 


No. 


cent 


1. 


Policy, Planning, Change, Innovation 


79 


10.7 


738 


10.7 


i280 


8.2 


2. 


Research 


71 


9.6 


791 


11.5 


3920 


9.8 


3. 


Design and Development 


44 


6.0 


82 


1.2 


2800 


7.0 




Dissemination and Communication 


29 


3.9 


316 


4.6 


920 


l.Z 


5. 


Evaluation 


40 


5.4 


109 


1.6 


3240 


8.1 


Subtotal 


263 


T C 


<:U3d 


Cd. D 


14160 


35.4 


6. 


Science and Technology 


5 


0.7 


49 


0.7 


600 


1.5 


/ . 


Data Analysis, Statistics, Other 


139 


18.9 


1127 


16.4 


2440 


6.1 




Analyses 




8.4 


147 


2.1 


800 


2.0 


8. 


Objectives and Criteria 


62 


9. 


Tests, Measurement, Data Collection 


65 


8.8 


617 


9.0 


5760 


14.4 




Instruments 












16.4 


10. 


Methodology, Technique, Theory 


79 


10.7 


1406 


20.4 


6560 


Subtotal 


350 


47.5 


3346 


48.6 


16160 


40.4 


11. 


Disciplines and Specialties 


11 


1.5 


614 


8.9 


520 


1.3 


12. 


Skills, Training, Professional 


28 


3.8 


169 


2.5 


2480 


6.2 




Education 














13. 


Personnel 


5 


0.7 


280 


4.1 


2240 


5.6 


14. 


Content and Media 


80 


10.9 


435 


6.3 


4400 


11.0 


Subtotal 


124 


16.9 


1498 


21.8 


9640 


24.1 .. 


Total 


737 


100.0 ' 


6880 


100.0 


39960 


99.9 
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Counts of this order for ERIC-like documents might be encountered in 
a retrospective search of the U.S. educational literature for, cay, only the 
past five years. If the search were to include a longer period or the litera- 
ture of other countries, the counts would be considerably highor and propor- 
tions among content categories might change. 

Table 4 suggests there is a rough "order of magnitude" difference in 
counts with approximately ten books and fifty to sixty journal articles and 
other documents for every one nonbook instructional material. (If a larger 
journal and another document base were assumed, e.g., a ten-year retrospec- 
tive search, the ratios would approach 1:10:100.) 

Fixing on the top cluster in Table 4, the "Pol icy . . .RDD&E" cluster, 
we find counts of 263 nonbook instructional materials, 2036 books, and an 
estimate of at least 14,160 journal articles. For this cluster the percen- 
tage for nonbook instructional materials are quite similar to the percentages 
for journals and other documents. By comparison, the percentage for books 
in the "Evaluation" and in "Design and Development" categories are markedly 
smaller. Our interpretation is that "Evaluation" and "Design and Develop- 
ment" are relatively new area*; not yet well-represented in "hard-cover" 
literature. The nonbook instructional materials collections contain a 
large proportion of semiformal instructional materials, which, like the 
ERIC documents, are more reflective of recent activity. 

The second cluster, "Science, Data, Objectives, Tests, Methodology," 
provides much of the "disciplined inquiry" technical content for the dis- 
ciplines of the previous clur^ter. The counts are similar, although a little 
higher: 350 nonbook instructional materials, 3,346 books, over 16,160 
journal articles and other documents, lite "methodology Technique and 
Theory" category and the "Data Analysis, Statistics, Other Analyses" 
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categories are the two major contributors. Taken together they account for 
over 250(1 books and 20n other instructional materials. The disnroportional ly 
higher percentage of nonbook instructional materials in the "Objectives and 
Criteria" category is explained primarily by the large number of "behavioral 
objectives" modules listed in Resourc e s for Performance-Based Education . 

The reader is urged to examine the three figures in Appendix A at this 
point. They present the scatter diagrams relating counts of instructional 
materials, ERIC descriptors, and book titles. The data have been plotted 
on log-log paper to make the counts more manageable. Lines of best fit 
are also displayed together with their regression equations. Points which 
are far off the regression lines indicate content categories which are not 
well predicted. For instance, in Figure Al , "Data Analysis, Statistics 
and Other Analyses" (7) and "Objectives and Criteria" (0) are above the 
lines, indicating that there are many more (nonbook) instructional materials 
than would be expected from the ERIC descriptor counts. Conversely, 
"Personnel" (13) plots far below the line, indicating far fewer instructional 
materials than would be expected from the ERIC descriptor counts. 

Figure A2, relating instructional materials to book titles, shows that 
"Design and Development" (3) and "Objectives and Criteria" (8) are farthest 
above the line of best fit, indicating more nonbook instructional materials 
than would be predicted from the count of book titles. "Personnel" (13) 
is very much below the Ifne of best fit, indicating far fewer instructional 
materials than would be suggested than the book count. (The author was 
surprised to find any instructional materials in this content category.) 

Figure A3, relating book titles to ERIC descriptors, shows there are 
far more book titles in the "Data Analysis, Statistics, and Other Analyses" 
category (7) than would be expected from the ERIC descriptor count. 

O 
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Figure 3 can be used to illustrate the difference in the two lines of best 
fit. The line of best fit for books p redicted from ERIC counts is the line 
with the lower slope (it intercepts the vertical book title axis at about 
52 books, for 1,000 ERIC descriptors). The "Data Analysis, Statistics, 
Other Analyses" (7) point is the point farthest above this line. But, 
the line of best fit in predicting ERIC descriptor counts from book title 
counts is the steeper line. (It intercepts the horizontal ERIC descriptor 
axis at about 1,900 descriptors, for 10 book titles). On this line it is 
point 11 ("Disciplines and Specialities") not 7, which is the point farthest 
above this line, indicating there are markedly fewer ERIC descriptors in 
the "Disciplines and Specialities" areas than were expected given the 
number of book titles. Note that points 3 ("Design and Development") 
and 5 ("Evaluation") fall below both lines of best fit indicating relative- 
ly more ERIC descriptors than book titles, no matter which is used as the 
predictor. 

Possibly remarkable is the fact that the correlations among these 
three sets of counts are quite comparable: .60 for instructional materials 
and ERIC descriptors; .61 for instructional materials and book titles, and 
.62 for book titles and ERIC descriptors. Given the grossness of our 
categories and the difficulties we experienced in equating content classi- 
fications, we were surprised that the correlations would be this high. 
All three correlations are significant, although the coeff iciencts of aliena- 
tion are high (-.80, compared to a zero correlation) the error of estimate 
is reduced by only about 20 percent. Although there is a significant 
relation among the three sets of count, the relationships are far from 
perfect. 

Note that the high count categories, 1. "Policy, Planning, Change and 
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Innovation", 2. "Research" and 9. "Tests, Measurement, and Data Collec- 
tion Instruments" are near the diagonal lines of best fit on all three 
figures. These three content categories are reasonably well predicted from 
any of the three information bases. 

If our interest is focused on predicting the quantities of nonbook 
instructional materials, we note from figure Al that the following are 
the categories best predicted by ERIC descriptors (those located cl(5est 
to less steep lines of best fit): 1. Policy, Planning, Change, Innovation, 
2. Research, 3. Design and Development, 4. Dissemination and Communication, 
5. Evaluation, 9. Tests, Measurements, Data Collection Instruments, 
10. Methodology, Technique and Theory, 11. Disciplines and Specialties, 
12. Skills, Training and Professional Education, and 14. Content and 
Media. This is a remarkably large number of categories. What is not 
predicted well (points located farther from the line of best fit) are the 
four categories; 6. Science and Technology, 7. Data Analysis, Statistics, 
Other Analyses, 8. Objectives and Criteria, and 13. Personnel. 

Turning to figure A2 we note that, in general, book title counts are 
much less effective predictors of nonbook instructional materials. The 
following content categories lie closest to the line of best fit: 1. Policy, 
Planning, Change, Innovation, 2. Research, 4. Dissemination and Communica- 
tion, 9. Tests, Measurement and Data Collectioft "trrstruments, 10. Methodology, 
Technique and Theory, and 12. Skills, Training and Professional Education. 
Not predicted as well from book titles are the content categories: 3. Design 
and Development, 5. Evaluation, 6. Science and Technology, 7. Data Analysis, 
Statistics and Other Analyses , 8. Objectives and Criteria, 11. Disciplines 
and Specialties, 13. Personnel, and 14. Content and Media. 
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In Table 5 we have extracted the count data from Table 4 for only 
eleven content categories* for the two instructional materials bases, books 
and nonbooks. Entries are displayed in descending order based on book counts. 

The reader may note that the counts for nonbook and book instructional 
materials are of roughly similar rank orders (the rank order correlation is 
.31). Table 5 is a simple matrix relating the dimensions of content and 
format. The data counts are undoubtedly underestimates; and could be 
raised by more diligent search; however, the gross proportionalities among 
counts would probably remain quite similar. Note that the top five content 
categories account for 80 percent of the books and 62 percent of the non- 
books. These are well-established, traditional, educational research areas. 
By contrast the three areas of "Dissemination and Communication," "Evalua- 
tion," and "Design and Development" account for less than nine percent of 

the books and only sixteen percent of the nonbook instructional materials. 

TABLE 5 

COUNTS FOR ELEVEN R&D SUBJECT MAHER CONTENT CATEGORIES 
FOR BOOK AND NONBOOK INSTRUCTIONAL MATERIALS 



Content Category 


Nonbooks 


Books 




Count 


Per- 
cent 


Count 


Per- 
cent 


Methodology, Technique, Theory 

Data Analysis, Statistics, Other Analyses. 

Policy, Planning, Change, Innovation . . . 
Tests, Measurement, Data Collection Instru. 

Dissemination and Communication 

Design and Development 


79 
139 
71 
79 
65 
80 
29 
62 
40 
44 
5 


n.4 

20.0 
10.2 
11.4 
9.4 
11.5 
4.2 
8.9 
5.8 
6.3 
0.7 


1406 
1127 
791 
738 
517 
437 
316 
147 
109 
82 
49 


24.2 
19.4 
13.6 
12.7 
10.6 
7.5 
5.4 
2.5 
1.9 
1.4 
0.8 


Totals 


693 


99.8 


5819 


100.0 



The three categories: "Disciplines and Specialties," "Skills, Training, 
Professional Education," and "Personnel" have been omitted because they 
are not of direct interest for instructional content. They are of interest 
for personnel and training information. 
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F. Other Dimensions 

The previous sections have focused primarily on attempts to provide 
quantitative estimates of R&D subject content by major formats, notably 
ERIC documents, books and nonbook instructional materials. 

When we consider format and level of formal izati on > we immediately note 
that all of the books are at the formal level. This is not true for instruc- 
tional materials. We turned to Resour ces for Performance-Based Education 
(Houston, et al . , 1973) to obtain estimates for specific formats and levels 
of formalization.* Regarding formats, we found the results shown on Table 6. 

TABLE 6 

FORMATS OF R&D RELEVANT TITLES IN 
PERFORMANCE-BASED EDUCATION 



Format 


No. 


Per- 
cent 




134 


42.3 




59 


18.6 




47 


14.8 


Slides, tapes and films trips .... 


44 


13.9 




12 


3.8 




8 


2.5 




7 


2.2 




6 


1.9 


Total 


317 


100.0 



We note that 61 percent of these items are priced, and only 39 percent 
are marked "consult distributor." We have taken this as a rough indication 
of availability, and have assumed that the nearly 200 priced items are 
"formal" and that the 123 irems marked "consult distributor" are still in 
the "somi formal" category. 



*The Guba and Gephart base consists entirely of semiformal materials. Their 
Table 1 (Guba and Gephart, 1970, p. 35) indicates that 87 percent of the 327 
items were either textual materials or written learning exercises. 
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G. Summary 

This report has attempted to examine the content and structure of the 
domain of R&D training resources. It began by locating instructional 
materials within a larger matrix which classified R&D resources by structure 
and levels of formalization. 

Examination of ERIC, Books in Prin t and three nonbook instructional 
materials collections indicated there is roughly an "order of magnitude" 
ratio for the estimated quantities in these three bases. By using these 
ratios (Figure 4), we made a graphic display in which the sizes of the areas 
on the figure indicates the estimated relative sizes of educational R&D 
instructional resources. This figure suggests that 90 percent of the R&D 
instructional resources are largely informal and fugitive. 



FIGURE 4 

ESTIMATED RELATIVE PROPORTIONS OF EDUCATIONAL R&D 
INSTRUCTIONAL MATERIALS BY LEVEL OF FORMALIZATION 
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Many of the materials at this level are developed for the use of 
individual instructors. Generally they are not published, refereed, or 
referenced. Perhaps 10 percent find their way into the semi formal level 
where they may appear in a journal article, monograph, or limited edition 
of a module of film. The item may be referenced in ERIC or in compendia 
such as Resources for Performance-Based Educc?tio n, Marketable items at 
this level may find a commercial publisher, but most are destined for 
''thin-market" distribution. 

Possibly one or two percent achieve "hard-cover" commercial publication 
statuG, and well over 95 percent of the materials at this level appear in 
print format. We estimate that only a small fractior of one percent of all 
the R&D resources are themselves subjected to the RftD cycle (development, 
field test, revision) before release for publication. Given the iata 
presented in Table 6, there is reason to believe that a substantial propor- 
tion of the "R&D cycle" instructional materials will appear as printed 
"modules" or in some media form. 

Generally, "accessibility" and "fiualUy" will be related to the vertical 
dimension of Figure 4. The Iiigher areas are ^Nujected to greater external 
review, editing^ commercial publication and marketing, and ultimately to 
R&D testing* As they pass these "quality" screens, they tend to becone more 
physically accessible and easier to locate in catalogues and indexes. 

Our studies suggest that the number of fully developed and tested educa- 
tional R&D instructional materials must be much less than lon items. There 
are perhaps as many as 1,000 items which have undergone minimal development, 
but have probably been subject to only informal testing or evaluation. 
After actually counting 6880 book titles, we estimate that possibly 10,00''^ 
books and textbooks exist which have >-easonably direct bearing on RfnD content 
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as v/e have defined it. We have no way of directly estimating the size of 
the informal and fugitive base in Figure 4. It is certainly very large 5nd 
many times the number of books and textbooks. 

These figures suggest there is a large amount of relevant material. 
In a sense there is, but this is true partly because educational R&D 
training content is so broadly based. As Table 5 shows, well over half of 
the content of our book counts is contained in these three major categories: 
10. Methodology, Technique and Theory; 7. Data Analysis, Statistics, 
Other Analyses and 9. Tests, Measurement, and Data Collection Instruments. 
Much of this content is derived from, and shared with, behaviorial and social 
science. 

By contrast only 3.3 percent of the book titles fall in the "Design 
and Development" or "Evaluation" categories. We were able to locate less 
than 200 books and other instructional materials in either of these categories. 

When we examine data by time periods, it appears that possibly the most 
accessible and established, but least timely, instructional materials resources 
are books. It usually requires time to develpp a discipline or subject 
content area to the stage where it is well represented by books and text- 
books. Yet the contents of ERIC, represr-nted by reports collected since 
1966 and journal articles cited since 1969, are more reflective of recent 
R&D activity. We made the assumption that ERIC data may thus better reflect 
potential "raw" instructional materials resources. The lines of best fit 
(Appendix A, Figures Al and A2) seem to confirm this assumption. Hence 
despite the preponderance of book titles in the more classical "educational 
research" content areas, it is ali:c useful to note (see Table 5) that the 
percentages of nonbook instructional materials are markedly larger than the 
percentages for books in tne areas of "Evaluation" and "Design and Develop- 
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ment." The last part of Anoendix C displays time trend data which suggests 
that the content and character of the domain of educational R&D information, 
at least as indexed by ERIC, is shifting. Because of the very large base 
in well-established RftD areas, these changes are relatively small, but 
nevertheless, quite apparent. 

We regret that time and resources have not permitted a more detailed 
examination of the domain of R&D training resources. Classification by 
intended target audience, level of difficulty, estimated quality, etc., 
is possible, but is well beyond the time and resources of this project. 

What we have succeeded in doing is to go beyond complete guesswork by 
analyzing and comparing several large information bases in order to arrive 
at data-based estimates regarding subject content for various formats, 
including books, nonbook instructional materials, and ERIC documents. 
Appendices B and D may also be h^^lpful in indicating where to look and 
how much one may find with respect to several hundred specific subject 
headings. 

We hope that this report has helped to sketch out the rough outlines 
of the actual quantities and proportions of what may be found. 
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APPENDIX A 

SCATTERGRAMS OF COUNTS FOR R&D CONTENT 

Figures A-1, A-2 and A-3 present the scatter diagrams and correlations between 

logarithms of counts for instructional materials, book titles, and ERIC Descriptors. 

Logarithmic transformations have been employed to reduce the data to manageable 
★ 

configurations* 

The formulas for the lines of best fit are: 
Figure A-1 

log nr. of instructional materials = -1.467 + .757 log nr. of ERIC descriptors 

log nr. of ERIC descriptors = 3.267 + .472 log nr. of instructional materials 

Figure A-2 

log nr. of instructional materials = 0.013 + .617 log nr. of book titles 

log nr. of book titles = 1.569 + .603 log nr. of instructional materials 

Figure A-3 

log nr. of book titles = -0.594 + .776 log nr. of ERIC descriptors 

log nr. of ERIC descriptors = 2.763 + .494 log nr. of book titles 



Forman S. Acton Analysis of Straight-line Data . New York: Dover Publications, 1959, 
p. 223. Professor Acton notes that data that are counts of populations, vital 
statistics, census data and the like are almost always improved by taking logs. 
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APPENDIX B 



LISTING OF DESCRIPTORS AND IDENTIFIERS AND 1 
THEIR FREQUENCY OF USE IN RIE/CIJE (TO DECEMBER 1972) 



Policy. Planning, Change and Innovation 



896 


Curriculum Planning 


2627 


Educational Change 


(681) 


Educational Development 


2)36 


Educational Innovation 


1901 


Educational Planning 


769 


Educational Policy 


336 


Financial Policy 


898 


Instructional Innovation 


44 


Language Planning 


18 


Neighborhood School Policy 


1009 


Planning 


223 


Pol icy 


569 


Policy Formation 


1773 


Program Planning 


2G2 


Public Policy 


1566 


Social Change 


96 


Social Planning 


15.123 




(15,804) 





1 

Source Lockheed Information Sciences, March, 1973. Numbers in parentheses 
indicate that the descriptor is counted in another category. local without 
parens is for all descriptors in list without parens. Total in parens includes 
counts for descriptors in parens. Counts in brackets are not listed elsewhere. 
These counts are based on a total of 128 thousand RIE and CUE documents. 
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Research 



76 


Action Research 




35 


Ayricultural Research Projects 


1648 


Behavioral Science Research 


193 


Classroom Research 




196 


Creativity Research 




401 


Curriculum Research 




180 


Economic 


Research 




3529 


Educational Research 




203 


Environmental Research 




181 


Facility Utilization Research 


44 


Interest 


Research 




1231 


Language Research 




137 


Library Research 




240 


Media Research 




182 


Medical 1 


Research 




168 


Methods Research 




206 


Operations Research 




44 


Poverty Research 




1230 


Reading Research 




3466 


Research 






435 


Research 


and Development 


Centers 


27 


Research 


and Instruction 


Units 


54 


Research 


Comnittees 




(168) 


Research 


Coordinating Units 


168 


Research 


Criteria 




456 


Research 


Designs 




24 


Research 


Directors 




91 


Research 


Libraries 




(1684) 


Research 


Methodology 




864 


Research 


Needs 




149 


Research 


Opportunities 




520 


Research 


Problems 




123 


Research 


Proposals 




1232 


Research 


Reviews 




(131) 


Research 


Skin<; 




(181) 


Research 


Tools 




(358) 


Research 


Utilization 




(181) 


Researchers 




452 


Scientific Research 




45 


Textbook 


Research 




18.230 








(20,933) 









Design & Development 



950 Curriculum Design 

4961 Curriculum Development 

216 Design 

307 Design Needs 

69 Design Preferences 

223 Development 

681 Educational Development 

385 Instructional Design 

292 Film Production 

580 Materials Development 

379 Program Design 

2064 Program Development 

55 Psychological Design Needs 

1052 Systems Approach 

275 Systems Development 

290 Teacher Developed Materials 

83 Textbook Preparation 



12,862 



Dissemination 



348 Information Center 

1212 Information Dissemination 

441 Information Needs 

279 Information Networks 

177 Information Seeking 

537 Information Services 

423 Information Utilization 

168 Research Coordinating Units 

358 Research Utilization 

(107) Technical Reports 

391 Use Studies 



4,334 



(4.441) 



Evaluation 



81 


Administrator Evaluation 

t \\A mill i«# %0t w 1 ^tmimmvi w 1 1 


130 


Counselor Evaluation 


357 


Course Eval uation 


(208^ 


Cri teri a 

V 1 1 V\« 1 1 M 


884 


Curriculum Evaluation 

I 1 IWmImIII haTMIViMVI W 1 1 


1394 


Fval ua ti on Cri iipri a 


1 084 


Fvflluaf'inn Mpf hnrfQ 
u V a 1 uGi u 1 v^i 1 l ie Of lUUd 


21 ? 


i.vGiiuaoiuii I J ccud 




F\/a1 Mri'f* 1 on TprhniniiPQ 

UvGI lUQw lUfl 1 Cl«llll IvfUCd 


146 


Faculty Evaluation 


63 


Formative Evaluation 


(216) 


Media Selection 


166 


Medical Evaluation 


f 581 ) 


pprformancp Cri teri a 


(181) 


Performance Specifications 


484 


Personnel Evaluation 


79 


Preschool Evaluation 


840 


Program Effectiveness 


4814 


Program Evaluation 


203 


Psychological Evaluation 


1221 


Student Evaluation 


71 


Summative Evaluation 


896 


Teacher Evaluation 


286 


Textbook Evaluation 


138 


Textbook Selection 


14,?52 




(16,138) 





Science & Technology 



109 Scientific Attitudes 

373 Scientific Concepts 

200 Scientific Enterprise 

114 Scientific Literacy 

182 Scientific Principles 

(452) Scientific Research 

668 Technological Advancement 

602 Technology 

523 Theories 



2,771 



(3,223) 



9 



Data Analysis and Processiyig» Statistics & Other Analysis 



54 


Analysis of Covariance 


701 


Analysis of Variance 


12 


Bayesian Statistics 


29 


Cluster Analysis 


155 


Comparative Statistics 


55 


Component Analysis 


617 


Correlation 


1002 


Data Analysis 


688 


Data Collection 


433 


Electronic Data 


157 


Input-Output Analysis 


872 


Interaction Process Analysis 


115 


Linear Programming 


190 


Logic 


492 


Mathematical Models 


142 


Multiple Regression Analysis 


26 


Nonparametric Statistics 


528 


Predictive Validity 


180 


Predictor Variables 


148 


Probability 


91 


Probability Theory 




Protue evaluation 


(199) 


Sampling 


161/ 


b ta ti s tica 1 Mna lysi s 


10 


Statistical Bias 


1432 


Statistical Data 


198 


Statistical Studies 


251 


Statistical Surveys 


251 


Statistics 


- 1 


Suppressor Variable 


647 


Systems Analysis 


88 


Tests of Significance 


62 


Transformations [Mathematics] 


(273) 


Validity 


11, 335 




(11 ,807) 





o 
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Objectives and Criteria 



218 


Affective Objectives 


223 


Cognitive Objectives 


208 


Criteria 


(142) 


Criterion Referenced Tests 


577 


Course Objectives 


79 


Measurement Goals 


376 


Norms 


915 


Objectives 


30 


Psychomotor Objectives 


581 


Performance Criteria 


181 


Performance Specifications 


282 


Training Objectives 


3,670 




(3.812) 
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Tests, Measurement, Data Collection Instruments 



14 


Affective Tests 


138 


Pretesting 


674 


Aptitude Tests 


70 


Pretests 


431 


Attitude Tests 


48 


Prognostic Tests 


202 


Cognitive Measurement 


69 


Projective Tests 


130 


Cognitive Tests 


367 


Psychological Testing 


247 


Comparative Testing 


213 


Psychological Tests 


159 


Community Surveys 


174 


Psychometrics 


142 


Criterion Referenced Tests 


46 


Q Sort 


50 


Critical Incident Method 


2713 


Questionnaires 


150 


Educational Testing 


90 


Scores 


44 


Forced Choice Technique 


191 


Scoring 


614 


Intelligence Tests 


34 


Scoring Formulas 


872 


Interviews 


211 


Screening Tests 


343 


Item Analysis 


3 


Semantic Differential 


17 


Item Banks 


143 


Sociometric Techniques 


25 


Item Sampling 


448 


Standardized Tests 


872 


Interaction Process Analysis 


2038 


Surveys 


707 


Measurement 


1012 


Test Construction 


(79) 


Measurement Goals 


536 


Test Interpretation 


1243 


Measurement Instruments 


1041 


Test Reliability 


1043 


Measurement Techniques 


869 


Test Results 


43 


Mental lests 


50 


Test Reviews 


4 


Multi-dimensional scales 


108 


Test Selection 


214 


Multiple Choice Texts 


1301 


Test Validity 


59 


Nonverbal Tests 


28 


Test Wiseness 


45 


Norm Referenced Tests 


1717 


Testing 


(376) 


Norms 


321 


Testing Problems 


112 


Objective Tests 


270 


Testing Programs 


393 


Observations 


1163 


Tests 


219 


Performance Tests 


92 


Television Surveys 


530 


Personality Assessment 


39 


Timed Tests 


300 


Personality Tests 


36 


True Scores 


180 


Post Testing 


273 


Validity 


401 


Predictive Ability [Testing] 


26,668 


337 


Predictive Measurement 




(180) 


Predictor Variables 


(27,303) 





o 

ERIC 



B-10 



Methodology, Technique and Theory 

106 Autoinstructional Methods 

366 Case Studies 

159 Community Surveys 

993 Cost Effectiveness 

381 Course Organization 

50 Critical Incident Method 

67 Critical Path Method 

134 Cybernetics 

1820 Decision Making 

109 Deduction Methods 

245 Documentation 

724 Educational Technology 

583 Educational Theories 

330 Experimental Groups 

440 Experiments 

43 Flow Charts 

557 Follow-up Studies 

12 Force Field Analysis 

168 Game Theory 

302 Inductive Methods 

821 Instructional Improvement 

102 Instructional Systems 

770 Instructional Technology. 

(872) Interaction Process Analysis 

1211 Interdisciplinary Approach 

449 Laboratory Experiments 

347 Laboratory Techniques 

1536 Learning Theories 

548 Longitudinal Studies 



216 


Media Technology 


118 


Medi ati on Theorv 


383 


Methodol oov 


2868 


Model s 


644 


Predi ction 


1755 


Problem Solving 


314 


Prooram Rudop ti nnc 


440 


Prooram Costs 


lf)84 

1 UO*T 


Rpsparrh Mpt hoHol nav 


181 


Research Tools 


199 


Sampling 


210 


Scientific Methodology 


161 


Search Strategies 


91 


Set Theory 


1068 


Simulations 


(143) 


Sociometric Techniques 


(2038) 


Surveys 


(647) 


Systems Analysis 


5198 


Teaching Methods 


813 


Teaching Procedures 


31 


Theoretical Criticism 


523 


Theories 


30,270 




(33,970) 
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Disciplines & Specialties 



277 Anthropology 

301 Educational Economics 

352 Educational Psychology 

240 Educational Sociology 

(724) Educational Technology 

85 Experimental Psychology 

367 Information Science 

(770) Instructional Technology 

113 Labor Economics 

278 Social Psychology 
421 Sociology 

2,434 



(3,928) 
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Sk^'^^s^ Training, and Professional Education 



502 


Counspl or Trai ni no 


Pin 


npri ^ 1 nn Ma kinn ^tillc 


649 


Inservi ce FHuration 

AII^Wl V 1 WW lU wU ^ 1 w 1 1 


643 


Institutes 






UZ70 


.InK Ti^aininn 

KJyJlJ 1 1 u 1 M 1 


202 
cue 
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L.UUwl UUWljf 1 1 Q III 111^ 




1 paHpr^hin Tiraininn 

LCuUC ldlll|.^ llullllIlM 


dOR 


riuiiu^ciiicii 0 ucvc 1 U}jiiidi c 


258 


Manaaement Educati on 


105 


Management Games 


316 


Negro Education 


332 


On-the-Job Training 


15 


Performance Based Teacher Education 


615 


Post Secondary Education 


440 


Professional Continuing Education 


661 


Professional Education 


458 


Professional Training 


15 


Research Apprenticeships 


131 


Research Skills 


64 


Skill Analysis 


864 


Skill Development 


91 


SkiV.s 


315 


Task Analysis 


1282 


Task Performance 


65 


Technical Writing 


672 


Training 


^82) 


Training Objectives 


750 


Training Techniques 


11,537 




(11.819) 
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Personnel 

20 Audiovisual Directors 

723 Change Agents 

27 Clinical Professors 

29 Counselor Selection 

293 Employment Patterns 

195 Employment Projections 

230 Employment Statistics 

334 Employment Trends 

224 Extension Agents 

(146) Faculty Evaluation 

18 Film Production Specialists 

116 Guidance Personnel 

25 Information Scientist 

370 Job Analysis 

114 Job Development 

115 Job Market 
755 Job Skills 
249 Labor Force 
260 Labor Market 

86 Labor Supply 

36 Labor Turnover 

964 Manpower Development 

734 Manpower Needs 

547 Manpower Utilization 

182 Media Specialists 

115 Negro Employment 

135 Negro Leadership 

31 Negro Population Trends 



356 Paraprofessional Personnel 

319 Personnel 
(484) Personnel Evaluation 

237 Personnel Needs 

182 Personnel Policy 

282 Personnel Selection 

675 Professional Personnel 
11 Psychometrists 

180 Researchers 
91 Scientific Manpower 
81 Scientific Personnel 

258 Scientists 

175 Staff Improvement 

313 Staff Utilization 
13 Systems Analysts 
79 Talent Development 
77 Talent Identification 
20 Talent Utilization 
[5689] Teacher Education 
[201] Teacher Selection 
[190] Teacher Supply and Demand 



10,276 
[(16,986)] 
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Content & Media 



2517 Audiovisual Aids 

574 Audiovisual Instruction 

143 Autoinstructional Programs 

411 Case Studies [Education] 

395 College Curriculum 

299 College Instruction 

1378 Computer Assisted Instruction 

980 Course Content 

1816 Decision Making 

314 Educational Games 

274 Experimental Curriculum 

969 Experimental Programs 

183 Experimental Schools 

267 Experimental Teaching 

298 Field Experience Programs 

(292) Film Production 

495 Instructional Films 

1100 Films 

305 Films trips 

6 Instructor Centered Television 

778 Mass Media 

47 Media Selection 

(216) Media Technology 

532 Multimedia Instruction 

1976 Programmed Instruction 

614 Program Materials 

262 Programed Texts 

54 Programed Units 

1068 Simulation 

101 Simulators 

651 Tape Recordings 

399 Teaching Machines 

540 Televised Instruction 

89 Television Curriculum 

192 Transparencies 

84 Textbook Bias 

170 Textbook Content 

124 Workbooks 

20.405 
(20.913) 



APPENDIX C 

ESTIMATION OF NUMBER OF DOCUMENTS IN THE RIF OR CUE 
BASED ON FREQUENCY OF USE OF DESCRIPTORS AND IDENTIFIERS 

Appendix B lists the counts for frequency of use of 379 selected 
descriptors and identifiers. The list is somewhat arbitrary. It could be 
pruned back to 100 or expanded to 500. We have suggested that, since the 
counts are made on a very large document base, the relative frequency of 
counts or sets of counts may be moderately stable and useful. For instance, 
while there are 3466 documents with the descriptor, "research," there are 
only 223 documents with the descriptor, "development." But if the "research" 
set of descriptors is compared with the "design and development" set, the 
total relative frequencies are 18.2 thousand to 12.9 thousand. 

The problem with these counts is that they index "information" within 
documents. Since several kinds of information, calling for different descrip- 
tors, is contained in the same document, we know that the number of documents 
will be substantially smaller than the usage counts. How much smaller? Is 
there any way to estimate numbers of documents, given available counts of 
frequency of use of descriptors? This appendix describes our approach to this 
problem. 

As an example. Table 1 indicates that 16 terms were classed under the 
heading, "policy, planning, change and innovation." There were 15.1 thousand 
uses of these 16 descriptors. This class accounts for 8.2% of the frequency 
counts for classes of descriptors listed in Table 1. 

Taken together, the 98 descriptors under "policy, planning, change and 
innovation," plus the "R,D,D&E," account for 35.4% of the descriptor use 
frequency counts. These 98 descriptors were used over 65 thousand time in 
describing an estimated 16,000 to 29,000 documents and journal articles in 
ERIC. 
O 
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The second ''science and methodology" cluster, ranging from science and 
technology through statistics, objectives, methodology, theory, etc., involves 
164 descriptors which were employed nearly 75 thousand times to describe an 
estimated 18,000 to 30,000 documents and journal articles. 

The final cluster, which includes documents describing disciplines, skills, 
training, professional education, personnel, content, and media, involved 117 
descriptors, which were applied nearly 45 thousand times to describe an 
estimated 11,000 to 18,000 documents and journal articles. 

Overall, we estimate that ERIC alone now contains at least 45,000 items to 
which the 379 descriptors listed in Appendix B have been applied. The "RDD&E" 
collection alone (involving the 82 descriptors listed in Appendix B) is 
estimated at 14,000 to 22,000 items. Undoubtedly, many of these items may 
be of questionable quality, but it is evident that there is a remarkably large 
pertinent literature reflected in the ERIC collection. Since ERIC does not 
usually access cofmiercially published items and probably misses a small 
fraction of other eligible items, our guess is that the number of pertinent 
items, published during the same period (approximately 1966-1972 for RIE and 
1969-1972 for CUE), may be ^0% to 20% higher than our estimates for the 
ERIC collection alone. 

We now turn to a description of our method for estimating the number of 
documents from descriptor counts, Lockheed Information Sciences provides 
periodic listings of RIE/CIJE descriptors and identifiers and their frequency 
of use. They are computer generated through their ERIC/DIALOG Retrieval System. 
The DIALOG system is capable of displaying counts for individual descriptors 
and sets of descriptors of nearly any degree of complexity. Hence the count 
of documents for a particular search, say all documents to which the descriptors 
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(research iir development or evaluation) and (personnel or training or 
professional education) have been applied, can be obtained directly. 
However, the number of possible combinations for the 379 terms we have 
selected is 379 factorial, an impossibly large number. 

We have thus used an indirect approach. Our first step was to establish 
the average number of descriptors and identifiers applied to appropriate 
samples of RIE/CIJE documents. 

Our ''population" consisted of 950 items that had been extracted from 
the ERIC base by applying sets of "RDD&E" terms as well as "personnel" and 
"training" terns in different combinations. Hence our population may be 
biased in not reflecting the entire ERIC file, but if there is a bias, the 
collection is more appropriate for our purposes. 

Systematic samples of 100 RIE and 100 CUE items were taken and the 
number of descriptors and identifiers assigned to each document was made. 
In making these counts, authors, agencies, countries, states, and dates were 
excluded. Our sample of RIE documents displayed a skewed distribution of 
descriptor counts. 

One document had only three descriptors and one had twenty descriptors. 

The mean was 9.5 with a standard deviation of 3.8 and a standard error of 

the mean of 0.38. These data suggest that we may expect with practical 

confidence to find an average of 9 or 10 descriptors assigned to the typical 

"RDD&E" or "personnel and training" document. 

The number of descriptors assigned to journal articles indexed in CUE 

1 

is significantly smaller, with a range of one to ten descriptors, a mean of 

6.1, d standard deviation of 3.9, and standard error of the mean of 0.20. We 

1 

The t-test for differences between RIE and CUE means is over 8.0; while only 
2.6 is needed at the .005 level (one tail). 

O 
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have reasonable confidence of finding an average of 5 1/2 to 6 1/2 descriptors 
pev^ CUE journal item. In scanning these 200 sample RIE/CIJE citations, our 
impression was that possibly less than half of the descriptors were on our 
list of 379. No accurate match was made. If a relatively large number of 
descriptors (say 30 or more) are involved or where a smaller number, say ten 
or more closely associated descriptors are employed (e.g.. Research, Scientific 
Research, Research Methodology, Methodology, Scientific Methodology, etc.), our 
advice would be to sum the descriptor use counts and divide by 4 if using RIE 

and divide by 3 if using CUE to estimate, quite conservatively, the total 

number of items. These divisors are very rough estimates of the average 

number of descriptors from the selected set applied to the same documents. 

Where the number of descriptors is smaller (but at least ten) and the descriptors 

are relatively unrelated, then a divisor of 3 for RIE and 2 for CUE may be more 
2 

accurate. These estimators will not be useful when only a small number of 
descriptors are employed or when a moderate number of descriptors with great 
variance in counts are involved. To illustrate, note that the number of documents 
with descriptors A or B or C, must be at least as large as the largest single 
count. For instance, Appendix B indicates that under the classification, "Design 
and Development,'' there are 950 uses of Curriculum Design, 4961 uses of Curriculum 
Development, and 216 uses of Design. We know that the number of documents using 
at least one of these descriptors must at least equal 4961. Dividing the total 
(6127) by a divisor of 2 or 3 makes no sense. Even for the 12,862 uses of 17 
descriptors in the "Design and Development" class there are problems. Given 
the very large count for curriculum development and relatively large counts for 
program development and systems approach, three relatively unrelated terms, we 

2 

Often the counts are available for CUE and RIE combined. Since the number 
of documents now in CUE and RIE are roughly equal, one might take averages: 
say using a divisor of 3 1/2 if a very large number of items are involved, and 
^ V2 for moderate number, and possibly 1 1/2 for a relatively small number. 

ERIC 
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might estimate the number of documents indexed by this class of descriptors 
to be at least 8,000 and possibly over 10,000. Obviously, application of a 
divisor of 2 or 3 would lead to serious underestimates. On the other hand, 
consider the "Skills, Training and Professional Education" cluster. Here, 
there are 28 descriptors or identifiers and none has a count that is as high 
as 11% of the total count. In this case, a rough estimate of at least 3,000 
(=11,819x3.5) and possibly 5,000 (=1 1 ,819r2,5) documents seems realistic 

To summarize, it is possible to estimate roughly the number of documents 
in ERIC from the counts of frequency of use of descriptors. When the number 
of descriptors involved is small or a few descriptors account for a relatively 
large proportion of the total count, the estimates call for considerable 
guesswork and judgment. Conversely, when the number of descriptors is 
relatively large and the proportion of the total count represented by one or 
a few descriptors with high frequency of use is snail, use of a divisor may 
be indicated. The value of the divisor may range between 1.5 and 4, depending 
on the total number of descriptors involved and the proportion of RIE to 
CUE documents. Since CUE documents tend to have an average of 6 descriptors 
and RIE documents average 9 1/2 descriptors, divisors of 2 or 3 will be more 
appropriate for CUE, while 3 or 4 may be more appropriate for RIE. These 
divisors simply represent the estimated average number of times descriptors 
from the selected set will be applied to the same document. When highly 
accurate estimates are required, the ERIC/ DIALOG or a similar system is 
needed. If the work must be done manually, one can use the ERIC Educational 
Documents Indexes and the CUE Annual Cumulations . Working from the In dexes 
and the Cumulations is practical if the number of descriptors is not large. 
When the number is large, we suggest that sampling strategies be employed to 
reduce the task of matching only samples of the possible combinations of 
descriptors. 

ERIC 
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It should be noted that content in different areas is being accessed by 
ERIC at different rates. Although this is partially due to ERIC Central 
priorities and Clearinghouse policies, it undoubtedly reflects differential 
rates of knowledge documentation. Time and resources have not permitted 
extensive study of this area, but the data in Table 1 illustrate some short- 
term trends, based on ERIC accessions in the last six months of 1972. 



TABLE 1 

ACCESSIONS AND DESCRIPTORS FOR ERIC JULY-DECEIVER, 1972 



Item 


July Count 


December Count 


Increase 


Rate 


RIE (ED#1) 


59.226 


65,620 


6,394 


10.8% 


CUE (EJ#) 


53,972 


62.751 


8,779 


16.3 


Total 


113,198 


128,371 


15,173 


13.4% 


"Research" 


18,525 


20,933 


2,408 


13.0 


"Design & Development" 


11,196 


12,862 


1,666 


14.9 


"Evaluation" 


13,328 


15,969 


2.641 


j9.8 


Educati onal Devel opmen t 


465 


681 


216 


46.5 



The reader is cautioned that data over a longer time period should be 
employed to study trends. The data displayed in Table 1 do, however, illustrate 
the following: (a) over 15 thousand items were referenced in the six-month 
period, (b) CUE items (journal references) are graving at a faster rate than 
RIE (technical reports) items, (c) the rate for the "research" group (See 
Appendix B for complete lists of descriptors.) is approximately the same as 
the total of RIE and CUE, (c) the "design and development" descriptor group 
shows only a very slightly higher rate, (3) the "evaluation" group shows 
a markedly higher rate, and (f) the individual descriptor, educational 
O'vel opmen t, shows a very high rate, but on a very small base. 
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The point of this analysis is to emphasize that estiniates of the amount 
of information available for specific content areas will be conditioned by 
time. Documents pertaining to educational development and almost the entire 
"evaluation" group of descriptors are being accessed at higher rates than the 
relatively more established "research" group. 

If there were more time, trends based on the past several years could 
have been developed and then used to make projections for a number of content 
areas* Such projections might be of use if content estimates were desired 
for several years hence. 
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ESTIMATION OF NUMBER OF BOOKS IN PRINT RELATED 
TO EDUCATIONAL RDD&E PERSONNEL AND TRAINING* 



Policy, Planning, Change and Innovation 

18 Community and School [Policy] 

67 Education— Finance 

81 Education 

10 Education, Elementary 

—Aims and Objectives/ 

—Curriculum 
70 Education, Higher 

--Aims and Objectives/ 

"Curriculum 
12 Education, Secondary 

--Aims and Objectives/ 

--Curriculum 
18 Education, Urban 
59 Educational Innovations 
40 Educational Planning 
30 Education and State 
17 Organizational Change 
27 Planning 
55 Political Psychology 
16 Politics and Education 
7 Public Opinion 
46 School Management 
139 Social Change 
21 Social Policy 
5 Social Science and State 



738 



*Estiinates based on counting selected book titles under subject headings 
listed on the following pages. Source is Subject Guide to Books in Print , 
1972, 
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Research 



6 


Economic Research 


42 


Education Cost 


108 


Educational Research 




Education — Research 


3 


Interest — Psychology 


68 


Languages — Psychology 


9 


Learning and Scholarship 


6 


Library Science Research 


31 


Linguistic Research 


22 


Marketing Research 


81 


Operations Research 


58 


Organi zational Research 


c 


Physical Education and Training--Research 


30 


Political Science Research 




Political Science--Research 


23 


Psychiatric Research 


30 


Psychological Research 


68 


Poverty Research 


25 


Reading--Psychology of 


43 


Research 


14 


Research Industrial 


4 


Research Libraries 


(16) 


Research Management 


80 


Social Science Research 


31 


Sociological Research 


2 


Sociology—Research 


2 


Textbooks 


791 




(807) 





ERIC 



D-3 



Design & DeveTopment 

6 Comniunications--Audio Visual Aids 
4 Design 
2 Education 

25 Education— Curricula 
(4) Filmstrips 
(17) Moving Pictures--Production and Direction 
( 9) Moving Pictures--Editing 

Photography (over 100) 
31 System Analysis 
9 Systems Engineering 
5_ Textbook Preparation 

82 

(112) 



Dissemination 



D-4 



61 
5 

16 
3 

15 
10 
14 
3 
4 
14 
8 
11 



32 
27 
53 
9 
2 
29 
(59) 
(27) 

316 

(402) 



Communication 

Communication--Social Agent 

Communication in Management 

Communication in Medicine 

Conmuni cation in Science/in Social Science 

Communication of Technical Information 

Communications Research 

Diffusion of Innovations 

Dissemination 

Educational Exchanges 

Information Storage and Retrieval 

Information Storage and Retrieval 

—Education/ 

--Political Science/ 

--Science/ 

--Social Science 
Information Services 
Report Writing 
Technical Writing 
Technology--Information Services 
Technology Transfer 
Technology and Civilization 
Educational Innovation 
Technical Innovation 



ERIC 



Evaluation 



6 Education (including Elementary, 

Higher, and Secondary) 

(2) Job Evaluation 

(10) Employees, Rating of 

(4) Executives, Rating of 

(19) Grading and Marking (Students) 

3 Occupational Training 

4 Organization Research 
2 Performance Criteria 

1 Performance Specifications 

2 School Management and Organization 

3 Student Evaluation of Curriculum 
13 Teachers, Rating of 

2 Textbooks 

7 Paperbound Titles 



28 


Evaluation- 


-Theory and Strategy* 


10 


Evaluation- 


-Curriculum & Instruction* 


8 


Evaluation- 


-Social* 


3 


Evaluation- 


-Guidelines* 


17 


Evaluation- 


-Causation & Impact* 


109 






(144) 







These category titles are not as listed in Books in Print s 



Science & Technology 



26 Science (out of approximately 500) 

(23) Science Methodology 

(7) Science, As a Profession 

23 Technology 

49 



(79) 



D-7 



Data Analysis and Processing, Statistics & Other Analysis 



3 

15 
3 

14 
2 
4 

(35) 
18 
18 
58 
28 
25 
(4) 
C>600] 
22 
37 
12 
1 
1 
8 

(n) 

3 
1 

60 
['^200J 
[=100] 
(13) 
19D 



Analysis of Variance 4 

Bayesian Statistical Decision Theory 16 

Cluster Set Theory 9 

Correlation (Statistics) 1 

Data Processing 10 
Decision Logic Tables [MOO] 

Decision Making—Mathematical Models 43 

Dynamic Programming 7 

Distribution (Probability Theory) (31) 

Econometrics (4) 

EconomicS"-Mathematical 5 

Educational Statistics 7 

Educational Surveys (IS) 

Electronic Data Processing 6 
Electronic Data Processing in Education 45 

Experimental Design [Statistics] 338 

Factor Analysis 15 

Factorial Experiment Designs 8 

Frequency Curves 7 

Interindustry Economics [Input-Output 12 

Analysis] 3 

Interaction Analysis in Education 45 

Interative Methods--Mathematics 20 

Latent Structure Analysis 3 
Linear Programming 

Logic 1127 
Matrices 

Mathematical Models [(>2540)] 
Mathematical Statistics/ 

--Bibliography 

--Programmed Instruction/ 

—Tables 



Monte Carlo Methods 
Multivariate Analysis 
Nonparametric Statistics 
Order Statistics 
Prediction--Psychology 
Probability 

Programming (Mathematics) 
Regression Analysis 
Sampling (Statistics) 
Scale Analysis (Psychology) 
Social Sciences--Statistics 
Social Sciences— Statistical Methods 
Social Surveys 

Statistical Hypothesis Testing 
Statistical Decision 
Statistics 

Stati sti CS--B i bl i og raphy 
Statistics--Charts , Tables 
Stati sti cs--Graphic Methods 
S ta t i s ti cs - - P rog rammed Instruction 
Sequential Analysis 
System Analysis (over 130) 
Time Series Analysis 
Transformations 



ERIC 



Objectives and Criteria 



89 Aims and Objectives 
2 Education 

5 Educatiors, Elementary 

--Aims and Objectives 
31 Education, Higher 

--Aims and Objectives 
11 Education, Secondary 

-rAims and Objectives 
1 Motor Ability Testing (of 23) 
5 Performance Criteria 
_3 Occupational Training (of 27) 



147 



Tests, Measurement) Data Collection Ins t rumen ts 



19 


Ahi 1 i f v — Tp^f i nn 


7 


Attitude Tests 


3 


Character Tests 


14 


Education Examinations. DLiPstions pfr 


107 


Educational Tests and MpasurMnpnts 


10 


Emoloveps. Ratina of 


29 


Examinations 


1 


Expruf* "i vPs — ^Ahi 1 "i tv Tpstino 


4 


Ex pr [J ti VPS— - Pa i"i nn of 


4 


npnpral FHurational rip\/p1 nnmont Tpctc 


19 


Gradina and Markino f Students) 




Intel 1 iaence Tpsts 

All \/\» 1 1 IMwIIWw IwOwO 


3 


I nt prvi pws 


17 


InterviewinQ 


11 


Interaction Anal vsi s in Educati on 


1 


Mathemati cal Abi 1 i tv- -Testi na 


7 


Measure Theory 


2 


Mprhaniral Ahilifv Tpsfirtn 


104 


Mpntal Tpsts 


2 


Motivation rPsvrholoav) Tpstinn 


5 


Motor Ahi 1 i tv Tpstina 

1 1 v/ U v/ 1 riLy III U Jr IC^UIIl^ 




Musir Ah"i 1 itv--Tps ti no 

1 luo 1 \tf r^u III ujf icouill^ 


1 


Observations ( Educational Mpthod) 

wl^Owl VUwIwIlO ^L*\JU^UvlVIIUi 1 IC U 1 Iw U / 


19 


Personality Assessment 


40 


Personal itv Tpcts 


10 


Prediction 

ri wuiwwiwii 


13 


Predi cti on of Schol as ti c Succpss 

r 1 WUIWwIwII \J t ^WIIwlUOwl^ wUW^C^O 




Proipctfnn Tpsts 

r 1 c\# tt 1 \jf 1 1 CO tt o 


71 


Psvchompt ri cs 

1 w 1 1 wl 1 Iw w 1 1 w O 


3 


Questioning 




Oups ti onnai rps 

v^uco ui\^iiiia 1 i CO 


A 
•t 


Rpa di nn Ahi 1 "i tv Tps t "i nn 


4 


Scale Anal vs i s ^ Ps vchol oov ) 

*j \A 1 w ni lOijrOio ^rojf\#iiv/i v^jr / 


7 


Scholastic Antit»"^ti Tpst 

ti^Xviiv/iuouiw niyui wvi«4iw icou 


2 


^pnt pncp PniTinl p ti on Tps t 
«j 1 1 oci i\#c 1 c 0 1 uii icou 


10 


Sociometrv 


1 


Surveys 


4 


Surveys, Educational 


2 


Surveys, Market 


34 


Surveys, Social 


3 


Testing 


1 


Vocabulary Tests 



D-lO 



Methodolopy, Technique, and Theory 



3 


Case Method 




11 


Cost Effectiveness 




29 


Critical Path Analysis 


23 


82 


Cybernetics 


45 


150 


Decision Making 


2 


12 


Documentation 


2 


52 


Education—Experimental Method 


10 


(37) Experimental Design 


9 


1 


Factorial Experimental Design 


10 


9 


Flow Charts 


9 


1 


Games, Theory of 
Human Behavior 


2 


83 


15 


12 


Human Engineering 


11 


14 


Induction (Logic) 


7 


(11) 


Interaction Process 


23 


200] 


Learning, Art of 


35 


101 


Learning, Psychology of 


12 




Logic 


10 


2 


Market Surveys 


21 




[Methodology--see under following 


31 




discipline headings] 


(4) 



24 
18 
52 
1 



Economics— Methodology 
Education—Experimental Methods 
HLffnanities--Methodology 
31 Political Science Methodology 
29 P sycho 1 ogy- -Methodol ogy 
26 Science- -Methodology [Selected from 

49 



Science- ^ 

Social Sciences--Methodology 
S oc i ol ogy - - Met h odo 1 ogy 



^75] 



2 
37 
10 
(45) 
212 

12 



[Model s--see under following discipline 

headings] 

Economic Development--Mathematical Models 
Economics— Mathematical Models 
Education— Mathematical Models 
Human Behavior— Mathematical Models 
ildrketing--Mathematical Models 
Mathematical Models 

Political Science— Mathematical Models 

Psychol ogy--Mathematical Models 

Social Psychology— Mathematical Models 

Social Sciences^-Mathematical Models 

Sociology- -Mathematical Models 

PERT (network analysis) 

Prediction 

Problem Solving 

Program Budgeting 

Psychol ogy--Experinients 

Psychology— Experimental 

Sampling 

Scale Analysis (Psychology) 
Searching--Bibl iographical 
Set Theory 
Simulation Method 
Sociometry 
System Analysis 
Teaching 
Value Analysis 



1406 



rC>1703)T 



D-n 



Disciplines & Specialties 



18 Of (=800) 




161 




22 




220 




(25) 


5 


(of 20 


11 


(of 55) 




10 




3 


10 


(of 42) 




3 


61 


of 315 


15 


of 122 


60 


of=700 




13 




614 



Economics 

Education and Sociology 
Educational Anthropology 
Educational Psychology 
Educational Statistics 
Information Sciences 
Political Psychology 
Political Science 
Political Sociology 
Psychol ogy--Experimental 
Technology--Education 
Social Psychology 
Social Sciences 
Sociology 
Technology 



ERLC 



D-12 



Skills, Training, and Professional Education 



1 


Education 


36 


Employees, Training of 


10 


Executives, Training of 


4 


Industrial Management 


4 


Interns (Education) 


2 


Job Analysis 


2 


Job Description 


2 


Job Evaluation 


18 


Job Satisfaction 


16 


Management, Study & Teaching/Case Studies 


18 


Management Games 


5 


Neg roes— Education 


28 


Occupational Training 


4 


Psychology, Industrial 


9 


Professional Education 


3 


Retraining, Occupational 


7 


Technical Education 


169 





ERIC 



D-13 



Personnel 

14 Agricultural Extension 
[Extension Agent] 

3 Educational Media Personnel 
2 Guidance Student 

12 Industrial Management 

2 Industrial Management--Research 
10 Industrial Sociology 

1 Job Vacancies 

n Labor Supply 

5 Labor Turnover 
12 Manpower Policy 
20 Organization 

10 Organizational Research 

6 Negroes Employment 

4 Occupational Mobility 
(28) Occupational Training 

12 Occupations 

9 Professional Education 

16 Professions 

25 Personnel Management 

3 Psychology, Applied 



40 Psychology, Industrial 
3 Psychology as a Profession 

1 Research as a Profession 

18 Research, Industrial Management 
7 Science as a Profession 

2 Scientists, Employment 
9 Scientists 

20 Social Groups 

3 Wome n — Emp 1 cyme n t 
[123] Teachers, Training of/ 

Inservice Training 
[ 6] Teacher Selection 
[ 4] Teacher Supply and Demand 



280 
[(441)] 



ERIC 



Content and Media 



25 Audiovisual Education 

22 Audiovisual Materials 

3 Case Method 

6 Coinmunication--Audio-Visual Aids 

1 Communication--Bibliography 
13 Computer Assisted Instruction 

1 13 Education--Curricula 

(49) Education--Experirnental Methods 

7 Education, Higher--Curricula 
12 Education, Primary--Curricula 
12 Education, Secondary--Curricula 
17 Educational Games 

2 Lantern Slides 

15 Moving Pictures in Education 

27 Programmed Instruction 

8 Radio in Education/in Adult Education 
(37) Simulation Method 

10 Sound— Recording and Reproducing 

59 Teaching— Aid and Devices 

8 Teaching Machines 

3 Technical Education 

30 Television in Education 

28 Textbooks 
10 Visual Aids 

4 Visual Education 



435 



(521) 



